sr/context: implement /contexts/{context}/... prefixed handlers#30189
sr/context: implement /contexts/{context}/... prefixed handlers#30189nguyen-andrew wants to merge 10 commits intoredpanda-data:devfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds Confluent-compatible context-prefixed Schema Registry routes (/contexts/{context}/...) so clients can target non-default contexts by setting a context-prefixed base URL. It does this by introducing request-rewriting helpers to scope subjects/query params and by updating authorization to evaluate ACLs against the context-qualified subject derived from the URL prefix.
Changes:
- Add
/contexts/{context}/...route registrations that rewrite requests and delegate to existing handlers. - Introduce context parsing/normalization and subject scoping utilities, plus unit/integration tests for routing and ACL isolation.
- Extend Schema Registry auth resource handling to support authorization against context-qualified subjects for prefixed routes.
Reviewed changes
Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/rptest/tests/schema_registry_test.py | Adds end-to-end tests for context-prefixed routing, delete-context normalization, serde client base-URL behavior, and ACL isolation. |
| tests/go/go-kafka-serde/go.mod | Updates Go toolchain/deps for the Go serde test client (used by the new serde acceptance test). |
| tests/go/go-kafka-serde/go.sum | Updates dependency lockfile for the Go serde test client. |
| src/v/utils/variant.h | Adds extend_variant_t helper to extend std::variant types. |
| src/v/utils/tests/variant_test.cc | Compile-only test coverage for extend_variant_t. |
| src/v/utils/tests/BUILD | Registers the new variant_test target. |
| src/v/pandaproxy/schema_registry/test/context_router.cc | Adds unit tests for context normalization and request rewrite helpers. |
| src/v/pandaproxy/schema_registry/test/BUILD | Registers the new context_router_test target. |
| src/v/pandaproxy/schema_registry/service.cc | Registers context-prefixed routes and delegates via rewrite wrappers; updates route resource typing. |
| src/v/pandaproxy/schema_registry/handlers.cc | Normalizes context path parameter when deleting contexts (bug fix). |
| src/v/pandaproxy/schema_registry/errors.h | Adds context_invalid error info helper. |
| src/v/pandaproxy/schema_registry/error.h | Adds context_invalid to the Schema Registry error_code enum. |
| src/v/pandaproxy/schema_registry/error.cc | Maps context_invalid to an HTTP 400 response. |
| src/v/pandaproxy/schema_registry/context_router.h | Implements context normalization and rewrite helpers for subject/path/query scoping. |
| src/v/pandaproxy/schema_registry/authorization.cc | Updates auth resource extraction to support context-prefixed subject authorization. |
| src/v/pandaproxy/schema_registry/auth.h | Introduces context_prefix_subject and route_resource to resolve route-time resources into auth-time resources. |
| src/v/pandaproxy/schema_registry/BUILD | Adds context_router.h and links the variant utility into schema registry. |
| src/v/pandaproxy/api/api-doc/schema_registry.json | Documents the new /contexts/{context}/... endpoints for compatibility. |
5eae7af to
866cdb4
Compare
|
Force push to address copilot comment |
Retry command for Build#83227please wait until all jobs are finished before running the slash command |
866cdb4 to
1bb9ae6
Compare
|
Force push to change confluent-kafka-go upgrade to v2.3.0. The only thing we need from the upgrade is the URL path fix (confluent-kafka-go#943, landed in v2.1.0). Jumping to v2.14.0 was overkill and pulled in unnecessary transitive deps (Azure SDK, JWT, oauth2). v2.3.0 is the first stable release past the fix. |
1bb9ae6 to
f3b2f7a
Compare
|
Force push to fold in a security fix. Snyk flagged 4 transitive vulns in tests/go/go-kafka-serde |
Introduces context_router.h with inline helpers that will be used by
context-prefixed route wrappers:
- normalize_context(): canonicalize a URL path context parameter
by stripping outer ':' delimiters, adding '.' prefix, and
rejecting embedded colons (400 Bad Request via new
context_invalid error code)
- starts_with_context(): detect subjects already qualified with
a context prefix
Aliases like "staging", ":.staging:", and ".staging" all resolve to
the canonical form ".staging".
Includes gtest coverage for both helpers.
Includes gtest coverage for all helpers.
Adds a type-level utility for extending a std::variant with additional alternative types without repeating the original type list. This enables composing variant types where a superset variant needs all alternatives from a base variant plus extras.
Splits the auth resource type into `route_resource` (route registration
time) and `resource` (authorization time). The new
`context_prefix_subject` variant in `route_resource` qualifies the
subject with the {context} path param before the ACL check runs, then
resolves to `context_subject`.
This ensures a user with ACLs on "foo" (default context) cannot access
:.staging:foo via the /contexts/.staging/subjects/foo/... URL.
Registers 15 context-prefixed routes that have a {subject} path
parameter. Each route extracts the context from the URL prefix,
scopes the subject with it via scope_subject_param(), and
delegates to the existing handler.
Covers: subject CRUD, versions, compatibility, config/{subject}, and
mode/{subject} — all via /contexts/{context}/... URLs.
Includes unit tests for scope_subject_param() and ducktape coverage
for all 15 endpoints and ACL isolation.
Registers 4 context-prefixed routes for /schemas/ids/{id} and its
sub-resources (/schema, /versions, /subjects). Adds a
scope_subject_query helper that injects the context as a "subject"
query parameter, scoping schema lookups to the specified context.
Includes unit tests for scope_subject_query and ducktape coverage
for schema lookup and sub-resource queries.
Adds scope_subject_prefix_query(), which injects or prepends the
normalized context into the subjectPrefix query parameter. The
context-prefixed GET /contexts/{context}/subjects route uses this
helper to scope subject listings to the specified context.
Includes unit tests for scope_subject_prefix_query and ducktape
coverage verifying subject isolation across contexts.
Registers context-prefixed routes for context-level config/mode and
schema types:
- GET/PUT/DELETE /contexts/{context}/config
- GET/PUT/DELETE /contexts/{context}/mode
- GET /contexts/{context}/schemas/types (pass-through)
Adds inject_context_as_subject(), which sets the subject path
parameter to a context-only qualified subject (e.g., ":.staging:").
The config and mode wrappers use this to delegate to the existing
config/mode subject handlers. Schema types are global so the context
is accepted for compatibility but ignored.
Includes unit tests for inject_context_as_subject and ducktape
coverage for all operations.
While implementing context-prefixed route handlers, noticed that
DELETE /contexts/{context} was not normalizing the context path
parameter before the default-context check.
Apply normalize_context() to the context path parameter before the
default-context check. This ensures alias forms like "staging",
":.staging:", and ".staging" all resolve to the same canonical context
for deletion.
Includes ducktape coverage cycling through alias forms.
f3b2f7a to
179c951
Compare
|
Force push to rebase on latest dev. |
The v2.0.2 Schema Registry client stripped the path component from URLs (confluent-kafka-go#943), preventing context-prefixed URLs like /contexts/.serde from working. Fixed in v2.1.0 via PR redpanda-data#950.
A serde client configured with schema.registry.url pointing to /contexts/.serde performs a full produce/consume round-trip. Verifies schemas are registered in the target context and isolated from the default context. Parametrized across Python, Go, and Java clients to cover all language ecosystems.
179c951 to
a921cd1
Compare
|
Force push to bump golang/protobuf from v1.5.3 to v1.5.4 to try to fix issues with the CI docker image. |
pgellert
left a comment
There was a problem hiding this comment.
I'll double-check that all endpoints are added, with the correct ACLs, etc., in the next round. I haven't done that yet. But it's looking great!
| return context_subject::from_string(sub); | ||
| }, | ||
| [](const auto& res) -> auth::resource { return res; }); | ||
| } |
There was a problem hiding this comment.
What happens to the authorization of deferred handler like handle_get_schemas_ids_id_authz here? I think at the moment these deferred handlers have some hardcoded assumptions about what exact path they are handling, which might break going forward. E.g. handle_get_schemas_ids_id_authz uses the get_schemas_ids_id nickname but now the nickname could depend on whether it's a context-prefixed endpoint or not I think.
Can you add a few tests around the context path + ACLs + audit logging integration as well please?
| result = self.sr_client.request( | ||
| "POST", | ||
| f"contexts/{ctx}/subjects/{subject}/versions", | ||
| headers=HTTP_POST_HEADERS, | ||
| data=schema_data, | ||
| ) | ||
| assert result.status_code == requests.codes.ok, ( | ||
| f"POST versions failed: {result.text}" | ||
| ) | ||
| schema_id = result.json()["id"] | ||
| assert schema_id == 1 |
There was a problem hiding this comment.
nit: we could avoid recreating these custom request calls if we instead refactor SchemaRegistryRedpandaClient slightly to allow take an explicit base path (that defaults to random.shuffle(list(self.redpanda.nodes))[0].account.hostname but can be overwritten to point to a context-prefixed path.
| req.param.set( | ||
| ss::sstring("subject"), | ||
| ss::sstring(fmt::format("/:{0}:{1}", nctx, sub))); |
There was a problem hiding this comment.
Does rewriting the path parameter have any unintended consequences? I can see this being brittle or having unintended consequences to the data we log in audit logs or the logic in pandaproxy::log_request, either now or in the future.
| routes.routes.emplace_back(wrap( | ||
| ss::httpd::schema_registry_json::ctx_get_schemas_types, | ||
| auth::level::publik, | ||
| acl_operation::read, | ||
| auth::none{}, | ||
| // Schema types are global — the handler ignores the context. Validate | ||
| // it anyway for consistency with other /contexts/{context}/... routes. | ||
| ctx_route( | ||
| [](ss::http::request&, const ss::sstring& ctx) { | ||
| normalize_context(ctx); | ||
| }, | ||
| get_schemas_types))); |
There was a problem hiding this comment.
Does this endpoint exist in Confluent SR? I suspect maybe not.
| auto ctx_str = parse::request_param<ss::sstring>(*rq.req, "context"); | ||
| auto ctx = context{ctx_str}; | ||
| auto ctx = context{normalize_context(ctx_str)}; |
There was a problem hiding this comment.
nit: we have a couple of places where we do parse::request_param<ss::sstring>(*rq.req, "context") and then call normalize_context. Should we extract this to a helper and reuse?
| result = self.sr_client.request( | ||
| "GET", | ||
| "subjects", | ||
| params={"subjectPrefix": f":{ctx}:"}, | ||
| ) |
There was a problem hiding this comment.
nit: self.sr_client.get_subjects(subjectPrefix=f":{ctx}:")
| } // namespace util::detail | ||
|
|
||
| template<typename Variant, typename... Extra> | ||
| using extend_variant_t = |
Implement
/contexts/{context}/...prefixed routes for the Schema Registry,allowing clients to target a non-default context by prefixing any endpoint URL
with
/contexts/{context}. This is a Schema Registry compatibilityfeature that lets serde clients point their
schema.registry.urlat acontext-prefixed base URL (e.g.
http://host:8081/contexts/.serde) andtransparently register/lookup schemas in that context.
Each context-prefixed route extracts the
{context}path parameter, rewritesthe request (scoping the subject, query param, or injecting a synthetic subject),
and delegates to the existing handler. Four URL-rewriting strategies cover all
endpoint shapes:
scope_subject_param— qualifies the{subject}path param with the contextscope_subject_query— injects/qualifies the?subjectquery paramscope_subject_prefix_query— injects/prepends context into?subjectPrefixinject_context_as_subject— sets subject to context-only form for config/modeAuthorization uses a new
context_prefix_subjectresource type that qualifiesthe subject with the context before the ACL check, so users cannot bypass
context isolation via the URL prefix.
Also fixes a bug where
DELETE /contexts/{context}was not normalizing thecontext path parameter before the default-context check.
Fixes CORE-15191
Backports Required
Release Notes
Features
/contexts/{context}/...prefixed URLs on allendpoints, allowing serde clients to target a non-default context by
configuring their base URL (e.g.
schema.registry.url=http://host:8081/contexts/.myctx).